Digital pathology color calibration and validation

ABSTRACT

Color calibration for digital pathology is provided. A standard glass slide is prepared with a specimen having zero or more stains. The specimen is scanned a first time using a hyperspectral imaging system to produce a first digital image having XYZ color values. The specimen is scanned a second time using a digital pathology imaging system to produce a second digital image having RGB color values. The first and second digital images are then registered against each other to align the digital image data. Individual pixels of the first and second images may be combined in the registration process so that the first and second digital images have substantially similar pixel sizes. A lookup table is generated to associate XYZ color values to RGB color values. Once the lookup table has been generated, it can be used to present RGB color on a display using the corresponding XYZ color.

RELATED APPLICATION

The present application is a continuation of U.S. patent applicationSer. No. 16/095,267, filed on Oct. 19, 2018, which is a national stageentry of International Patent App. No. PCT/US2017/028532, filed on Apr.20, 2017, which claims priority to U.S. Provisional Patent App. No.62/325,330, filed on Apr. 20, 2016, which are all hereby incorporatedherein by reference as if set forth in full.

BACKGROUND Field of the Invention

The present invention generally relates to digital pathology and morespecifically relates to systems and methods for calibrating colormanagement in a digital pathology system and validating digital colordata in connection with corresponding physical color spectrum.

Related Art

Accurate communication of color information is a continuing challenge inmany industries. The most common approach to tackling this challengingproblem is to generate specially constructed test targets that representa particular color. In the photography industry, for example, thesespecially constructed test targets include an array of rows and columns,with each cell having a constant color. For reflected lightapplications, the specially constructed test target comes in the form ofpaper and for transmitted light applications, the specially constructedtest target comes in the form of film. For example, for naturephotography, the color patches are spectrally matched to blue sky, greenfoliage, brown skin, and so forth.

In the digital pathology industry, a significant problem exists in thatthe color being presented on a display is not the same as the color ofthe stained specimen. Exacerbating this problem is the lack ofavailability of any specially constructed test targets. This isgenerally because paper and/or film is an inadequate substitute fortissue as a medium for carrying the stains that contain the colorinformation. Attempts to create such test targets using materials toapproximate tissue have been largely unsuccessful. For example, specialbiopolymer strips have been hand crafted and stained with standardpathology stains. However, hand crafted fabrication of these targets iscomplex and expensive. Moreover, the specially constructed biopolymerstrips, when stained, typically fail to provide an accurate color matchto that of the same stain as applied to tissue. Accordingly, theunsolved problem is how to construct a test target that approximatestissue and provides an accurate color match when stains are applied tothe test target.

Therefore, what is needed is a system and method that overcomes thesesignificant problems found in the conventional systems as describedabove.

SUMMARY

To solve the problems discussed above, described herein are systems andmethods for calibrating a digital pathology slide scanning system sothat the colors of a scanned specimen presented on a display aresubstantially the same as the colors of the stained or unstainedspecimen on the physical slide.

The inventor has recognized that because the spectral properties of thestaining agent change when bound to tissue, a test target that isconstructed to faithfully approximate the characteristics of tissue withor without stains will always suffer significant drawbacks. This isbecause a proper solution requires that the spectral properties of thetest targets match those of the objects being imaged. Accordingly, inthe present description the inventor provides a solution that employsone or more standard slides as test targets that are used tosuccessfully calibrate an imaging system.

In one aspect, a standard slide is prepared with a specimen. A singlestain or a combination of stains or no stains can be applied to thespecimen. A grid is overlayed or superimposed on the slide to divide thespecimen into discrete sections. A digital image of the specimen on theslide is then obtained using a hyperspectral imaging system (referred toherein as the “hyperspectral image” or the “H image” or the “HYP image”or the “XYZ image”). Hyperspectral imaging systems are typically imagetiling systems and, for example, a single field of view of thehyperspectral imaging system can be captured for each cell in the gridoverlay. The resulting hyperspectral image for each cell includes astack of images ranging between 400 nm and 750 nm (the visual spectrum).The number of images in the stack can vary, for example one image per 10nm separation. The hyperspectral image stack is then processed to resultin an XYZ color image having a plurality of individual picture elements(“pixels”) where each pixel has an XYZ color value. The hyperspectralimage is then registered to the grid by mapping the top left cornerpixel to the top left corner of the grid. The individual pixels in anXYZ color image can be combined to create super pixels, whichadvantageously reduces pixel location errors when subsequentlyassociating XYZ pixels created by the hyperspectral imaging system toRGB pixels created by the digital pathology system.

Next, the same slide with the grid overlay is scanned using a digitalpathology system having a color imaging capability. The resultingdigital image (referred to herein as the “pathology image” or the “Pimage” or the “PATH image” or the “RGB image”) has a red, green and blue(“RGB”) value for each pixel. The pathology image is then registered tothe grid by mapping the top left corner pixel to the top left corner ofthe grid. The individual pixels in the RGB can also be combined tocreate super pixels and to match the pixel size of the hyperspectralimage to allow for direct color comparison between the XYZ values andthe RGB values. Also, the pixel sizes of both the hyperspectral imageand the pathology image can be downsized or upsized to optimize thepixel size matching.

Next, a lookup table (“LUT”) associating the XYZ color information for asingle pixel in the hyperspectral image and the RGB color informationfor the same pixel in the pathology image is generated. The LUTassociates the XYZ and RBG color information for all pixels in thepathology image. Advantageously registration of the hyperspectral imageto the pathology image, including image pixel size mapping, results in aone-to-one pixel association in the LUT. However, it is also possible toinclude in the LUT an average RGB value from a combined number of pixelsof the pathology image in association with the XYZ value from a singlepixel in the hyperspectral image or vice versa.

Once the LUT has been generated, it can be used by a display module sothat the colors of a scanned specimen in a digital image file having RGBcolor data can be presented on a display using the correspondinghyperspectral XYZ color to result in the displayed color beingsubstantially the same as the colors of the specimen on the physicalslide. The displayed color can be measured by a color measurement devicesuch as a colorimeter or a spectrophotometer.

In an alternative embodiment, a standard slide having a specimen isscanned using the hyperspectral imaging system as described above togenerate the XYZ image. The same slide is also scanned using the digitalpathology system to generate the RGB image. The pixels of the XYZ imageand RGB image are registered to each other to align the respectiveimages. If the size of the individual pixels of the imaging sensors inthe hyperspectral imaging system and the digital pathology imagingsystem differ, then pixel combining or downsizing may be employed tofacilitate proper alignment of the XYZ and RGB images and proper imagepixel size matching.

After scanning, one of the XYZ or RGB images is indexed to identify asmall number of colors into which every pixel in the image being indexedcan be assigned while also minimizing the error values for eachassociation of an image pixel to a color. For example, while the camerasensor may be capable of sensing millions of colors, the indexingprocess may advantageously reduce the number of colors in the XYZ imageto ten and each pixel in the XYZ image is assigned to one of the tencolors. In one embodiment, during indexing all pixels that are close tothe same color are averaged into a single color of the index colorpalette. This process is done iteratively until all pixels have beenassigned to an averaged single color value where the error value of theoriginal pixel color compared to the averaged single color is minimizedacross all pixels in the XYZ image. In one embodiment, the error valuecan be minimized using a root mean square analysis.

Once the XYZ image has been indexed, the result is a set of N pixelgroups, where N is the index value (ten in the example above) andcombining the pixels in the N pixel groups results in the complete XYZimage. A pixel grouping is also referred to herein as an “index.” Afterthe XYZ and RGB images have been registered, the pixels in the RGB imagecan be likewise associated into the same N pixel groups in accordancewith the pixel registration between the XYZ and RGB images. Each of theN pixel groups from the RGB image are analyzed to calculate an averagecolor value for each of the N pixel groups. The result is that eachindex in the XYZ image has an average color value and each index in theRGB image has an average color value and these average color values areused to generate the LUT.

Once the LUT has been generated, it can be used by a display module sothat the colors of a scanned specimen in a digital image file having RGBcolor data can be presented on a display using the correspondinghyperspectral XYZ color to result in the displayed color (e.g., asmeasured by a colorimeter or spectrophotometer) being substantially thesame as the colors of the specimen on the physical slide.

Other features and advantages of the present invention will become morereadily apparent to those of ordinary skill in the art after reviewingthe following detailed description and accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The structure and operation of the present invention will be understoodfrom a review of the following detailed description and the accompanyingdrawings in which like reference numerals refer to like parts and inwhich:

FIG. 1 is a plan view diagram illustrating an example digital pathologyslide having a barcode and a grid overlay according to an embodiment ofthe invention;

FIG. 2 is a block diagram illustrating an example prior arthyperspectral imaging system according to an embodiment of theinvention;

FIG. 3A is a block diagram illustrating an example hyperspectral imagingstack as generated by a hyperspectral imaging system according to anembodiment of the invention;

FIG. 3B is a graph diagram illustrating an example color matching foruse with a hyperspectral imaging stack according to an embodiment of theinvention;

FIG. 3C is a block diagram illustrating an example hyperspectral imagein XYZ color according to an embodiment of the invention;

FIG. 3D is a flow diagram illustrating an example process for convertinga hyperspectral image stack to a single XYZ color image according to anembodiment of the invention;

FIG. 4A is a block diagram illustrating an example image processordevice according to an embodiment of the invention;

FIG. 4B is a block diagram illustrating example superpixels of ahyperspectral image in XYZ color according to an embodiment of theinvention;

FIG. 5 is a flow diagram illustrating an example process for calibratingcolor values generated by a digital pathology scanning apparatus using asuperpixel process according to an embodiment of the invention;

FIG. 6 is a flow diagram illustrating an example process for validatingcolor values generated by a digital pathology scanning apparatus using asuperpixel process according to an embodiment of the invention;

FIGS. 7A, 7B and 7C are a graph diagrams illustrating examplecomparisons of color values of a specimen scanned by a hyperspectralimaging system versus superpixeling color values of the same specimenscanned by a digital pathology imaging system and presented on a displayand measured by a color measurement device according to an embodiment ofthe invention;

FIG. 8 is a block diagram illustrating an example set of pixel groupsthat form a composite XYZ image according to an embodiment of theinvention;

FIG. 9 is a flow diagram illustrating an example process for calibratingcolor values generated by a digital pathology scanning apparatus usingan indexing process according to an embodiment of the invention;

FIG. 10 is a flow diagram illustrating an example process for validatingcolor values generated by a digital pathology scanning apparatus usingan indexing process according to an embodiment of the invention;

FIGS. 11A, 11B and 11C are graph diagrams illustrating examplecomparisons of color values of a specimen scanned by a hyperspectralimaging system versus indexing color values of the same specimen scannedby a digital pathology imaging system and presented on a display andmeasured by a color measurement device according to an embodiment of theinvention;

FIG. 12A is a block diagram illustrating an example processor enableddevice 550 that may be used in connection with various embodimentsdescribed herein;

FIG. 12B is a block diagram illustrating an example line scan camerahaving a single linear array;

FIG. 12C is a block diagram illustrating an example line scan camerahaving three linear arrays; and

FIG. 12D is a block diagram illustrating an example line scan camerahaving a plurality of linear arrays.

DETAILED DESCRIPTION

Certain embodiments disclosed herein provide for color calibration of adigital pathology system. For example, one embodiment disclosed hereinallows for a pathology slide to be scanned by a hyperspectral imagingsystem and the colors of the resulting digital hyperspectral imagecompared to the colors of a digital image of the same slide scanned bythe digital pathology system. The comparison results in a lookup tablethat translates RGB values into XYZ values so that when the digitalslide image scanned by the digital pathology system is presented on adisplay, the presented colors match the XYZ values corresponding to thetrue colors of the physical specimen on the slide. After reading thisdescription it will become apparent to one skilled in the art how toimplement the invention in various alternative embodiments andalternative applications. However, although various embodiments of thepresent invention will be described herein, it is understood that theseembodiments are presented by way of example only, and not limitation. Assuch, this detailed description of various alternative embodimentsshould not be construed to limit the scope or breadth of the presentinvention as set forth in the appended claims.

FIG. 1 is a plan view diagram illustrating an example digital pathologyslide 10 having a barcode 20 and a grid overlay 30 according to anembodiment of the invention. In the illustrated embodiment, the gridoverlay 30 is positioned over the sample that is on the slide 10. Thegrid overlay 30 is used to facilitate registration of the hyperspectraldigital image to the digital pathology digital image. An alternative wayto register the images is to use image pattern matching of the twodigital images in order to align the hyperspectral digital image and thedigital pathology digital image. For successful image pattern matchingit is helpful to have digital images of the same magnification.

FIG. 2 is a block diagram illustrating an example prior arthyperspectral imaging system 50 according to an embodiment of theinvention. In the illustrated embodiment, the hyperspectral imagingsystem 50 includes a 2D pixel array monochrome camera 60, a microscope70, a slide 80 that supports a specimen and a narrow band filter wheel90. In one embodiment, the monochrome camera 60 of the hyperspectralimaging system 50 is a monochrome line scan camera. Preferably, themonochrome line scan camera 60 in the hyperspectral imaging system 50has the same characteristics as the color line scan camera used in thedigital pathology imaging system. Employing line scan cameras having thesame characteristics (e.g., pixel size) in both the hyperspectral anddigital pathology imaging systems advantageously simplifies imageregistration by reducing or eliminating the need for pixel matching—orby allowing the pixels to be easily resampled, e.g., by downsampling thepixels into larger superpixels to simplify registration of the XYZ andRGB images.

FIG. 3A is a block diagram illustrating an example hyperspectral imagingstack 100 as generated by a hyperspectral imaging system according to anembodiment of the invention. As will be understood by the skilledartisan, a hyperspectral imaging system generates a set of individualimages that are each captured using a different wavelength of light, forexample by using a narrow band filter wheel and capturing an image ofthe same region using each filter of the filter wheel. This set ofimages is referred to herein as a hyperspectral stack or a spectralstack. A spectral stack is can be generated for an individual region ofa sample or for an entire sample/entire slide. For example, in oneembodiment, a hyperspectral imaging system using a line scan camera cancapture a whole slide image using each filter on the filter wheel togenerate a whole slide image spectral stack.

FIG. 3B is a graph diagram illustrating an example color matching foruse with a hyperspectral imaging stack according to an embodiment of theinvention. In the illustrated embodiment, color matching functions canbe applied to each digital image in the whole slide image spectral stackat a variety of wavelengths of light (110, 120 and 130) to generate awhole slide hyperspectral image 140 in XYZ color. FIG. 3C is a blockdiagram illustrating an example whole slide hyperspectral image 140 inXYZ color according to an embodiment of the invention.

FIG. 3D is a flow diagram illustrating an example process for convertinga whole slide hyperspectral image stack 100 to a single hyperspectralimage 140 in XYZ color according to an embodiment of the invention.Initially, whole slide images for each different wavelength of light arecaptured by the hyperspectral imaging system to generate a hyperspectralimage stack 100. Then color matching functions are applied to eachdigital image in the spectral stack 100 at a variety of wavelengths oflight (110, 120, 130) to generate a whole slide hyperspectral digitalimage 140 of the entire specimen in XYZ color.

FIG. 4A is a block diagram illustrating an example image processordevice 260 according to an embodiment of the invention. In theillustrated embodiment, the image processor device 260 is processorenabled device having a processor 267 and a non-transitory data storagearea 265 for storing information and instructions that can be executedby the processor 267. For example, the data storage area 260 may store aplurality of hyperspectral XYZ images and a plurality of digitalpathology RGB images and a plurality of instructions for processing suchimages. As shown in the illustrated embodiment, the image processordevice 260 includes a register module 270, a superpixel module 280, anindex module 290 and a LUT module 295. In one embodiment, the superpixelmodule 280 and the index module 290 may be combined into a color module285. The image processor device 260 may also be communicatively coupledwith an integral or external display device 576. In one embodiment, acolor measurement device 577 may be configured to read color informationfrom the display device 576 and translate the color information into oneor more XYZ values.

The register module 270 is configured to register two digital images toeach other. For example the register module 270 is configured toregister a hyperspectral XYZ image to a digital pathology RGB image. Theregister module 270 registers two digital images by aligning the imagedata of the two images in X-Y to bring the two images into X-Yalignment, for example by pattern matching of features in the imagedata. The register module 260 also registers two digital images byadjusting the digital images so that they have common characteristics.For example, the register model 270 may evaluate and adjust imagecharacteristics including image pixel size and spatial alignment oftranslation, rotation, and magnification. Additionally, the registermodule 270 may also account for optical distortions between the twoseparate systems, which can be detected at the pixel level.

The superpixel module 280 is configured to identify contiguous imagepixels that have the same or similar color values and combine thosepixels into a single superpixel. The superpixel module 280 is alsoconfigured to determine a color value for the superpixel by averagingthe color values of all of the individual image pixels in thesuperpixel. Averaging is important to reduce noise due to possiblemeasurement and registration errors. The average color value can bedetermined for a superpixel, for example, by summing the color values ofthe image pixels in the superpixel and dividing the sum by the number ofpixels for that superpixel. The superpixel module 280 can advantageouslyidentify a plurality of superpixels in a digital image and determine acolor value for each of the plurality of superpixels.

The index module 290 is configured to identify individual image pixelsthat have the same or similar color values and assign these individualimage pixels to one of a plurality of color indices. The index module290 is also configured to determine a color value for each color indexby averaging the color values of all of the individual image pixels inthe respective color index. Averaging is important to reduce noise dueto possible measurement and registration errors. The average color valuecan be determined for an index, for example, by summing the color valuesof the image pixels in the index and dividing the sum by the number ofpixels for that index.

The LUT module 295 is configured to generate one or more lookup tablesthat correlate XYZ color values to RGB color values.

FIG. 4B is a block diagram illustrating an example registered wholeslide image 170 according to an embodiment of the invention. In theillustrated embodiment, whole slide image 170 may be a hyperspectralimage or a digital pathology image. During the registration process,image data from the hyperspectral digital image and the pathologydigital image is analyzed to achieve X-Y alignment. For example, theimage data may be analyzed to identify features in the image data thatcan be matched and aligned in order to register the hyperspectraldigital image to the pathology digital image in X-Y.

Advantageously, pattern matching can be used to associate commonfeatures between the hyperspectral digital image to the pathologydigital image to facilitate X-Y alignment.

Additionally during the registration process, the image data for thehyperspectral digital image and the pathology digital image is convertedto common characteristics. This is because the imaging hardware of thehyperspectral scanning system and the pathology scanning system isunlikely to produce identical digital image data with respect to, forexample, the magnification and image pixel size in the digital imagedata. Accordingly, during the registration process, the image data forthe hyperspectral digital image and the pathology digital image isadjusted to have common characteristics. For example, a magnificationadjustment may be needed and an adjustment to a common image pixel sizeis nearly always needed. In the illustrated embodiment, image pixel 180is an image pixel having the common image pixel size.

FIG. 5 is a flow diagram illustrating an example process for calibratingcolor values generated by a digital pathology scanning apparatus using asuperpixel process according to an embodiment of the invention. Certainsteps of the illustrated process may be carried out by an imageprocessor device such as previously described with respect to FIG. 4A.Initially, in step 300 one or more test slides are prepared. A testslide is any type of slide that will be scanned by both thehyperspectral imaging system and the digital pathology imaging systemand used for calibration/validation purposes. Accordingly, no specialtype of slide preparation is required to be a test slide and there areno special characteristics of a test slide. Any slide having any stainscan be used as a test slide to calibrate a digital pathology scanningapparatus. This is a substantial deviation from all prior colorcalibration attempts and eliminates the problems of attempting to createa test slide or a test color sample. Using a slide prepared in thenormal way with a specimen and stains also allows the scanning apparatusto be calibrated using actual stains as modified by their application totissue that will be encountered during production scanning. Thisprovides a significant advantage. Moreover, using multiple test slidescan result a combined LUT that includes color values for multiplestains.

Additionally, in one embodiment, a registration grid can be overlayed onthe slide—preferably over the portion of the slide that includes thesample. The registration grid, if present, can be later used as a markerin the hyperspectral digital image and the digital pathology digitalimage to register the hyperspectral digital image to the digitalpathology digital image by aligning the image data in X-Y.

Next, in step 310 a hyperspectral image is scanned and stored. Thehyperspectral image may be scanned as separate image tiles using tilingsystem hardware or the hyperspectral image may be scanned as a wholeslide image using line scanning system hardware. Advantageously, thepresent color calibration and validation systems and methods arehardware agnostic.

After scanning, the native hyperspectral image includes one or morespectral stacks having a plurality of individual images that are eachprocessed using color matching functions to generate a single digitalimage in XYZ color. Next, in step 320 a color digital pathology image isscanned and stored. The scanned digital pathology image is in RGB color.The digital pathology image may also be scanned as separate image tilesusing a tiling system or the digital pathology image may be scanned as awhole slide image using a line scanning system. While there areadvantages to using cameras with the same or very similarcharacteristics such as pixel size and pixel number in the hyperspectralimaging system and the digital pathology imaging system, theseadvantages primarily serve to simplify the image registration processand make the registration process robust.

Next in step 330, the hyperspectral image and the digital pathologyimage are registered to each other. Image registration includes X-Yalignment, for example by pattern matching, and conversion to commoncharacteristics, for example, magnification and image pixel size. Insome embodiments, the registration process includes generation ofresampled image pixels (larger or smaller) to make the individual imagepixel size from the separate scanning systems substantially the same. Insome embodiments, the registration process may also include localizedvariations in translation to account for optical distortions.

Once the hyperspectral image and the digital pathology images areregistered to each other, in step 340 the color groups in thehyperspectral image are determined. In a simple embodiment, eachindividual image pixel in the hyperspectral image is its own colorgroup. However, this would result in significant noise because anindividual image pixel is very small and consequently the sample sizefor each color is also very small. To reduce the noise, in step 340contiguous individual image pixels having the same or very similarcolors are combined into larger superpixels such as superpixel 190 shownin FIG. 4B. The larger the superpixel, the greater the sample size,which has the advantage of reduced noise. However, a disadvantage ofincluding more pixels in a superpixel is that it reduces the range ofcolors across all superpixels. Once a superpixel has been identified,the color of the superpixel determines a color group. In one embodiment,the color values of all image pixels included in the superpixel areaveraged together to determine an average color value and that averagecolor value is determined to be the color for that color group, as shownin step 350.

Advantageously, the color groups in the hyperspectral image cover everyimage pixel in the hyperspectral image and each color group has an X-Yperimeter. Accordingly, because the hyperspectral image and the digitalpathology image have been registered to each other, the X-Y perimetersof the color groups from the hyperspectral image can be applied to thedigital pathology image as shown in step 360 to associate the individualimage pixels of the digital pathology image into the same color groupsas the hyperspectral image. Accordingly, the color values of theindividual image pixels of each color group in the digital pathologyimage can be similarly averaged in step 370 to determine an averagecolor value for each color group in the digital pathology image.

Once the average color values for the various color groups in thehyperspectral image and the average color values for the same colorgroups in the digital pathology image have been established, these colorvalues can be correlated to each other in a lookup table having XYZcolor values associated with their correlated RGB color values, as shownin step 380. In one embodiment, the lookup table can be embedded in adata structure that houses the digital pathology image. In oneembodiment, the correlation of the XYZ color data to the RGB color datacan be included in the digital pathology image data structure as part ofan International Color Consortium (ICC) profile. For example, aspreviously mentioned, the lookup table can be embedded in the digitalpathology image data structure or alternatively, the information in thelookup table can be converted into a mathematical model or formula or aset of executable instructions and the model or formula or set ofinstructions can be embedded in the digital pathology image datastructure. An advantage of embedding the model or formula or set ofinstructions is that the data size of the model or formula or set ofinstructions is smaller and thereby reduces the size of the digitalpathology image data structure. Another advantage is that the model orformula or set of instructions functions to average out minordifferences and discrepancies in the correlation of the XYZ color datato the RGB color data, which may be introduced, for example, due tometamerism. Metamerism is when two colors that are not actually the same(i.e., they reflect different wavelengths of light) appear to be thesame under certain lighting conditions.

In one embodiment, a single combined lookup table is generated over timefrom a plurality of slides having a plurality of different stains.Advantageously, a single combined lookup table can be generated andoptimized over time such that the single combined lookup table can beused for any type of digital pathology slide with any type of stainingprofile.

FIG. 6 is a flow diagram illustrating an example process for validatingcolor values generated by a digital pathology scanning apparatus using asuperpixel process according to an embodiment of the invention. Certainsteps of the illustrated process may be carried out by an imageprocessor device such as previously described with respect to FIG. 4A.In the illustrated embodiment, a test slide is initially prepared instep 400. As previously discussed, the test slide can be any slideprepared in the normal fashion using a specimen and zero or more stains.Next, in step 410 a lookup table is generated, for example using theprocess previously described with respect to FIG. 5 . The lookup tablemay contain color values such as those shown in the Hyperspectral XYZcolumn and the associated Digital Pathology RGB column of Table 1 below,where each row represents the same color group (e.g., a superpixel) inthe hyperspectral digital image and the digital pathology digital image.

TABLE 1 HYPERSPECTRAL DIGITAL DIGITAL SUPERPIXEL XYZ PATHOLOGY RGBPATHOLOGY XYZ 1 0.777, 0.631, 0.551 225, 147, 169 0.770, 0.625, 0.556 20.743, 0.574, 0.500 219, 130, 154 0.750, 0.570, 0.510 3 0.712, 0.426,0.454 213, 117, 140 0.719, 0.431, 0.450 4 0.683, 0.485, 0.413 208, 105,128 0.678, 0.481, 0.418

Table 1 illustrates a correlation of hyperspectral image data in XYZcolor values to digital pathology image data in RGB color values todigital pathology image data in XYZ color values.

Next, in step 420 the XYZ values of the digital pathology image aredetermined. This can be done in at least two ways. A first way is thatthe XYZ values for the digital pathology image can be calculated fromthe RGB values of the digital pathology image—for example using a lookuptable or a formula. A second way is that the digital pathology image canbe presented on a display and the colors emitted from the display can bemeasured using a color measurement device (e.g., colorimeter orspectrophotometer) that measures color in XYZ value.

Finally, in step 430 the calculated or measured XYZ value for aparticular region (e.g. a superpixel) of the digital pathology image iscompared to the hyperspectral image XYZ value for the same region. Inthis fashion, the color information generated by digital pathologyapparatus and presented on a display screen can be validated.

FIGS. 7A, 7B and 7C are a graph diagrams illustrating examplecomparisons of color values of a specimen scanned by a hyperspectralimaging system versus superpixeling color values of the same specimenscanned by a digital pathology imaging system according to an embodimentof the invention. The XYZ color values from the digital pathologyimaging system were obtained by presenting the digital slide image on adisplay and measuring a superpixel region with a color measurementdevice. Alternatively, because a superpixel region can be very small andtherefore difficult to measure with a color measurement device, thecolor value for the superpixel may be used to fill up the entire displaywith a single color and then measuring a region with a color measurementdevice. As demonstrated by the graph, the color values measured ascoming off of the display screen were extremely close to the true colorvalues as measured by the hyperspectral imaging system, with an averagedifference of 2.18, which is less than one Just Noticeable Difference.

In FIG. 7A, graph 200 shows a comparison of the system hyperspectralvalue for lightness to the digital pathology display value forlightness. Similarly, in FIG. 7B, graph 220 shows a comparison of thesystem hyperspectral value for green/red to the digital pathologydisplay value for green/red. Similarly, in FIG. 7C, graph 240 shows acomparison of the system hyperspectral value for blue/yellow to thedigital pathology display value for blue/yellow. Additionally, in eachof the graphs 200, 220 and 240, it is evident that there are a verylarge number of individual comparisons. Notably, each individualcomparison corresponds to a separate superpixel. After the registrationprocess, although the number of image pixels in the hyperspectral imageand the digital pathology image is also very large, the consequence ofhaving large number of superpixels is that the sample size (i.e., numberof image pixels) for each superpixel is small and therefore noise isincreased in the data set.

FIG. 8 is a block diagram illustrating an example set of pixel groupsthat form a composite XYZ image 650 according to an embodiment of theinvention. In the illustrated embodiment, there are ten indices, namelyindex 1, 600, index 2, 605, index 3, 610, index 4, 615, index 5 620,index 6 625, index 7 630, index 8 635, index 9 640 and index 10 645.Each of the indices represents a separate color group for the underlyingdigital image. When combined together, the ten indices result in thecomposite image 650. The index color palette 660 represents each of thecolor values corresponding to each individual index.

As can be seen in the individual index images, each index imagerepresents a scattering of all image pixels in the underlying digitalimage that have the same color value within a certain threshold. Theindexing process can be applied to either the XYZ digital image or theRGB digital image. Advantageously, the indexing process analyzes thecolor values for each pixel in the digital image and identifies allpixels, regardless of X-Y location that belong to a single color value.In the illustrated embodiment, an entire digital pathology digitalimage, whether created by a hyperspectral imaging system or a pathologyimaging system can be indexed into about ten (10) color values. Asignificant advantage of indexing all image pixels in a digital imageinto a relatively small number of color values is that the sample sizefor each color value is significantly increased, which significantlyreduces noise. Another advantage of indexing all image pixels in adigital image into a relatively small number of color values is that itprovides the widest range of average color values from the smallestnumber of indices.

FIG. 9 is a flow diagram illustrating an example process for calibratingcolor values generated by a digital pathology scanning apparatus usingan indexing process according to an embodiment of the invention. Certainsteps of the illustrated process may be carried out by an imageprocessor device such as previously described with respect to FIG. 4A.Initially, in step 700 a slide is obtained. As previously discussed, anytype of slide with a specimen having zero or more stains is suitable.Next, in step 710 a hyperspectral image is scanned and stored as an XYZimage. As previously discussed, the hyperspectral image may be scannedas separate image tiles using any type of scanning system hardware.Next, in step 720 a color digital pathology image is scanned and storedas an RGB image.

Next in step 730, the hyperspectral image and the digital pathologyimage are registered to each other as previously described. Imageregistration includes X-Y alignment and conversion to commoncharacteristics such as image pixel size. Once the hyperspectral imageand the digital pathology images are registered to each other, in step740 the hyperspectral image is indexed to identify a set of colors intowhich every image pixel in the hyperspectral image can be allocated. Ina simple embodiment, the indexing process receives an index value (i.e.,the total number of indices) and then sorts the individual pixels intothat number of color groups in a fashion that minimizes the error valuesassociated with allocating each pixel into an index that is defined by acolor value that is not identical to the color value of the respectivepixel being allocated to the index. For example, the index module 290may be configured to use an index value of ten or fifteen or twenty forany digital image. In a more complex embodiment, the index module 290may be configured to analyze the digital image data to determine anoptimal index value that allocates each pixel into the smallest numberof indices that minimizes the error value over the entire digital image.

Once all of the individual image pixels have been allocated to an indexof the hyperspectral image, the color value of the respective index isdetermined by averaging the color values of all individual image pixelsin the respective index to determine an average color value and thataverage color value is determined to be the color for that respectiveindex of the hyperspectral image, as shown in step 750.

Advantageously, the combined indices in the hyperspectral image includeevery image pixel in the hyperspectral image. Accordingly, because thehyperspectral image and the digital pathology image have been registeredto each other, each index from the hyperspectral image can be applied tothe digital pathology image in step 760 in order to group together thesame individual image pixels included in an index for the hyperspectralimage in a corresponding index for the digital pathology image. This canbe accomplished because the hyperspectral image and the digitalpathology image were previously registered to each other and theirrespective image pixel sizes were adjusted to be the same.

Once all of the individual image pixels of the digital pathology imagehave been allocated to an index, the color value of each respectiveindex of the digital pathology image is determined by averaging thecolor values of all individual image pixels in the respective index todetermine an average color value and that average color value isdetermined to be the color for that respective index, as shown in step770.

Once the average color values for the various indices in thehyperspectral image and the average color values for the same indices inthe digital pathology image have been established, these color valuescan be correlated to each other in a lookup table having XYZ colorvalues associated with their correlated RGB color values, as shown instep 780. In one embodiment, the lookup table can be embedded in a datastructure that houses the digital pathology image. In one embodiment,the XYZ color data can be included in the digital pathology image datastructure as part of an International Color Consortium (ICC) profile. Aspreviously mentioned, the lookup table or mathematical model or formulaor set of instructions can be embedded in the digital pathology imagedata structure.

As previously described, a single combined lookup table canadvantageously be generated over time from a plurality of slides havinga plurality of different stains. Advantageously, a single combinedlookup table can be generated and optimized over time such that thesingle combined lookup table can be used for any type of digitalpathology slide with any type of staining profile.

FIG. 10 is a flow diagram illustrating an example process for validatingcolor values generated by a digital pathology scanning apparatus usingan indexing process according to an embodiment of the invention. Certainsteps of the illustrated process may be carried out by an imageprocessor device such as previously described with respect to FIG. 4A.Initially, a test slide is prepared in step 800. As previouslydiscussed, the test slide can be any slide prepared in the normalfashion using a specimen and zero or more stains. Next, in step 810 alookup table is generated, for example using the process previouslydescribed with respect to FIG. 9 . The lookup table may contain colorvalues such as those shown in the Hyperspectral XYZ column and theassociated Digital Pathology RGB column of Table 2 below, where each rowrepresents a single color value (i.e., index) in the hyperspectraldigital image and the digital pathology digital image.

TABLE 2 HYPERSPECTRAL DIGITAL DIGITAL INDEX XYZ PATHOLOGY RGB PATHOLOGYXYZ 1 0.702 0.656 0.646 225 208 225 0.700 0.652 0.649 2 0.654 0.5940.619 218 197 219 0.648 0.583 0.618 3 0.745 0.716 0.677 230 217 2300.744 0.713 0.678 4 0.891 0.921 0.764 248 247 248 0.900 0.931 0.778 50.564 0.494 0.556 206 180 208 0.566 0.489 0.560 6 0.603 0.558 0.591 211192 215 0.606 0.553 0.596 7 0.787 0.780 0.707 235 226 236 0.789 0.7800.710 8 0.832 0.844 0.735 240 235 241 0.837 0.845 0.741 9 0.457 0.4240.506 184 168 197 0.460 0.419 0.509 10 0.382 0.324 0.450 174 147 1850.389 0.319 0.450

Table 2 illustrates a correlation of hyperspectral image data in XYZcolor values to digital pathology image data in RGB color values todigital pathology image data in XYZ color values.

Next, in step 820 the XYZ values of the digital pathology image aredetermined. As previously discussed, this can be done by calculating theXYZ values for the digital pathology image based on the RGB values ofthe digital pathology image or by presenting the color value on theentire display and measuring the color emitted from the display using acolor measurement device that measures color in XYZ value.

Finally, in step 830 the calculated or measured XYZ value for aparticular color value (e.g. an index) of the digital pathology image iscompared to the hyperspectral image XYZ value for the same index. Inthis fashion, the color information generated by digital pathologyapparatus and presented on a display screen can be validated against thetrue color as measured by the hyperspectral imaging system.

FIGS. 11A, 11B and 11C are a graph diagrams illustrating examplecomparisons of indexed color values of a specimen scanned by ahyperspectral imaging system versus indexed color values of the samespecimen scanned by a digital pathology imaging system according to anembodiment of the invention. The XYZ color values from the digitalpathology imaging system were obtained by presenting each indexed colorvalue on the entire display and measuring a portion of the display witha color measurement device. As demonstrated by the graphs, the colorvalues measured as coming off of the display screen were extremely closeto the true color values as measured by the hyperspectral imagingsystem, with an average difference that is less than one Just NoticeableDifference.

In FIG. 11A, graph 210 shows a comparison of the system hyperspectralvalue for lightness to the digital pathology display value forlightness. Similarly, in FIG. 7B, graph 230 shows a comparison of thesystem hyperspectral value for green/red to the digital pathologydisplay value for green/red. Similarly, in FIG. 7C, graph 250 shows acomparison of the system hyperspectral value for blue/yellow to thedigital pathology display value for blue/yellow. Additionally, in eachof the graphs 210, 230 and 250, it is evident that there are a verysmall number of individual comparisons. Notably, each individualcomparison corresponds to a separate index. Advantageously, having asmall number of indices results in a large number of pixels in eachindex, which consequently reduces the noise in the data set. When FIGS.11A, 11B and 11C are compared to FIGS. 7A, 7B and 7C, there are fewermeasurements, but the scatter due to noise is much less. Thisdemonstrates the advantage of having a very large number of image pixelsin each index that form the basis for determining the average color.

FIG. 12A is a block diagram illustrating an example processor enableddevice 550 that may be used in connection with various embodimentsdescribed herein. Alternative forms of the device 550 may also be usedas will be understood by the skilled artisan. In the illustratedembodiment, the device 550 is presented as a digital imaging device(also referred to herein as a scanner system or a scanning system) thatcomprises one or more processors 555, one or more memories 565, one ormore motion controllers 570, one or more interface systems 575, one ormore movable stages 580 that each support one or more glass slides 585with one or more samples 590, one or more illumination systems 595 thatilluminate the sample, one or more objective lenses 600 that each definean optical path 605 that travels along an optical axis, one or moreobjective lens positioners 630, one or more optional epi-illuminationsystems 635 (e.g., included in a fluorescence scanner system), one ormore focusing optics 610, one or more line scan cameras 615 and/or oneor more area scan cameras 620, each of which define a separate field ofview 625 on the sample 590 and/or glass slide 585. The various elementsof the scanner system 550 are communicatively coupled via one or morecommunication busses 560. Although there may be one or more of each ofthe various elements of the scanner system 550, for simplicity in thedescription that follows, these elements will be described in thesingular except when needed to be described in the plural to convey theappropriate information.

The one or more processors 555 may include, for example, a centralprocessing unit (“CPU”) and a separate graphics processing unit (“GPU”)capable of processing instructions in parallel or the one or moreprocessors 555 may include a multicore processor capable of processinginstructions in parallel. Additional separate processors may also beprovided to control particular components or perform particularfunctions such as image processing. For example, additional processorsmay include an auxiliary processor to manage data input, an auxiliaryprocessor to perform floating point mathematical operations, aspecial-purpose processor having an architecture suitable for fastexecution of signal processing algorithms (e.g., digital signalprocessor), a slave processor subordinate to the main processor (e.g.,back-end processor), an additional processor for controlling the linescan camera 615, the stage 580, the objective lens 225, and/or a display(not shown). Such additional processors may be separate discreteprocessors or may be integrated with the processor 555.

The memory 565 provides storage of data and instructions for programsthat can be executed by the processor 555. The memory 565 may includeone or more volatile and persistent computer-readable storage mediumsthat store the data and instructions, for example, a random accessmemory, a read only memory, a hard disk drive, removable storage drive,and the like. The processor 555 is configured to execute instructionsthat are stored in memory 565 and communicate via communication bus 560with the various elements of the scanner system 550 to carry out theoverall function of the scanner system 550.

The one or more communication busses 560 may include a communication bus560 that is configured to convey analog electrical signals and mayinclude a communication bus 560 that is configured to convey digitaldata. Accordingly, communications from the processor 555, the motioncontroller 570, and/or the interface system 575 via the one or morecommunication busses 560 may include both electrical signals and digitaldata. The processor 555, the motion controller 570, and/or the interfacesystem 575 may also be configured to communicate with one or more of thevarious elements of the scanning system 550 via a wireless communicationlink.

The motion control system 570 is configured to precisely control andcoordinate XYZ movement of the stage 580 and the objective lens 600(e.g., via the objective lens positioner 630). The motion control system570 is also configured to control movement of any other moving part inthe scanner system 550. For example, in a fluorescence scannerembodiment, the motion control system 570 is configured to coordinatemovement of optical filters and the like in the epi-illumination system635.

The interface system 575 allows the scanner system 550 to interface withother systems and human operators. For example, the interface system 575may include a user interface to provide information directly to anoperator and/or to allow direct input from an operator. The interfacesystem 575 is also configured to facilitate communication and datatransfer between the scanning system 550 and one or more externaldevices that are directly connected (e.g., a printer, removable storagemedium) or external devices such as an image server system, an operatorstation, a user station, and an administrative server system that areconnected to the scanner system 550 via a network (not shown). In oneembodiment, a color measurement device 577 may be configured to readcolor information from the user interface 575 and translate the colorinformation into one or more XYZ values.

The illumination system 595 is configured to illuminate a portion of thesample 590. The illumination system may include, for example, a lightsource and illumination optics. The light source could be a variableintensity halogen light source with a concave reflective mirror tomaximize light output and a KG-1 filter to suppress heat. The lightsource could also be any type of arc-lamp, laser, or other source oflight. In one embodiment, the illumination system 595 illuminates thesample 590 in transmission mode such that the line scan camera 615and/or area scan camera 620 sense optical energy that is transmittedthrough the sample 590. Alternatively, or in combination, theillumination system 595 may also be configured to illuminate the sample590 in reflection mode such that the line scan camera 615 and/or areascan camera 620 sense optical energy that is reflected from the sample590. Overall, the illumination system 595 is configured to be suitablefor interrogation of the microscopic sample 590 in any known mode ofoptical microscopy.

In one embodiment, the scanner system 550 optionally includes anepi-illumination system 635 to optimize the scanner system 550 forfluorescence scanning. Fluorescence scanning is the scanning of samples590 that include fluorescence molecules, which are photon sensitivemolecules that can absorb light at a specific wavelength (excitation).These photon sensitive molecules also emit light at a higher wavelength(emission). Because the efficiency of this photoluminescence phenomenonis very low, the amount of emitted light is often very low. This lowamount of emitted light typically frustrates conventional techniques forscanning and digitizing the sample 590 (e.g., transmission modemicroscopy). Advantageously, in an optional fluorescence scanner systemembodiment of the scanner system 550, use of a line scan camera 615 thatincludes multiple linear sensor arrays (e.g., a time delay integration(“TDI”) line scan camera) increases the sensitivity to light of the linescan camera by exposing the same area of the sample 590 to each of themultiple linear sensor arrays of the line scan camera 615. This isparticularly useful when scanning faint fluorescence samples with lowemitted light.

Accordingly, in a fluorescence scanner system embodiment, the line scancamera 615 is preferably a monochrome TDI line scan camera.Advantageously, monochrome images are ideal in fluorescence microscopybecause they provide a more accurate representation of the actualsignals from the various channels present on the sample. As will beunderstood by those skilled in the art, a fluorescence sample 590 can belabeled with multiple florescence dyes that emit light at differentwavelengths, which are also referred to as “channels.”

Furthermore, because the low and high end signal levels of variousfluorescence samples present a wide spectrum of wavelengths for the linescan camera 615 to sense, it is desirable for the low and high endsignal levels that the line scan camera 615 can sense to be similarlywide. Accordingly, in a fluorescence scanner embodiment, a line scancamera 615 used in the fluorescence scanning system 550 is a monochrome10 bit 64 linear array TDI line scan camera. It should be noted that avariety of bit depths for the line scan camera 615 can be employed foruse with a fluorescence scanner embodiment of the scanning system 550.

The movable stage 580 is configured for precise XY movement undercontrol of the processor 555 or the motion controller 570. The movablestage may also be configured for movement in Z under control of theprocessor 555 or the motion controller 570. The moveable stage isconfigured to position the sample in a desired location during imagedata capture by the line scan camera 615 and/or the area scan camera.The moveable stage is also configured to accelerate the sample 590 in ascanning direction to a substantially constant velocity and thenmaintain the substantially constant velocity during image data captureby the line scan camera 615. In one embodiment, the scanner system 550may employ a high precision and tightly coordinated XY grid to aid inthe location of the sample 590 on the movable stage 580. In oneembodiment, the movable stage 580 is a linear motor based XY stage withhigh precision encoders employed on both the X and the Y axis. Forexample, very precise nanometer encoders can be used on the axis in thescanning direction and on the axis that is in the directionperpendicular to the scanning direction and on the same plane as thescanning direction. The stage is also configured to support the glassslide 585 upon which the sample 590 is disposed.

The sample 590 can be anything that may be interrogated by opticalmicroscopy. For example, a glass microscope slide 585 is frequently usedas a viewing substrate for specimens that include tissues and cells,chromosomes, DNA, protein, blood, bone marrow, urine, bacteria, beads,biopsy materials, or any other type of biological material or substancethat is either dead or alive, stained or unstained, labeled orunlabeled. The sample 590 may also be an array of any type of DNA orDNA-related material such as cDNA or RNA or protein that is deposited onany type of slide or other substrate, including any and all samplescommonly known as a microarrays. The sample 590 may be a microtiterplate, for example a 96-well plate. Other examples of the sample 590include integrated circuit boards, electrophoresis records, petridishes, film, semiconductor materials, forensic materials, or machinedparts.

Objective lens 600 is mounted on the objective positioner 630 which, inone embodiment, may employ a very precise linear motor to move theobjective lens 600 along the optical axis defined by the objective lens600. For example, the linear motor of the objective lens positioner 630may include a 50 nanometer encoder. The relative positions of the stage580 and the objective lens 600 in XYZ axes are coordinated andcontrolled in a closed loop manner using motion controller 570 under thecontrol of the processor 555 that employs memory 565 for storinginformation and instructions, including the computer-executableprogrammed steps for overall scanning system 550 operation.

In one embodiment, the objective lens 600 is a plan apochromatic (“APO”)infinity corrected objective with a numerical aperture corresponding tothe highest spatial resolution desirable, where the objective lens 600is suitable for transmission mode illumination microscopy, reflectionmode illumination microscopy, and/or epi-illumination mode fluorescencemicroscopy (e.g., an Olympus 40X, 0.75NA or 20X, 0.75 NA).Advantageously, objective lens 600 is capable of correcting forchromatic and spherical aberrations. Because objective lens 600 isinfinity corrected, focusing optics 610 can be placed in the opticalpath 605 above the objective lens 600 where the light beam passingthrough the objective lens becomes a collimated light beam. The focusingoptics 610 focus the optical signal captured by the objective lens 600onto the light-responsive elements of the line scan camera 615 and/orthe area scan camera 620 and may include optical components such asfilters, magnification changer lenses, etc. The objective lens 600combined with focusing optics 610 provides the total magnification forthe scanning system 550. In one embodiment, the focusing optics 610 maycontain a tube lens and an optional 2× magnification changer.Advantageously, the 2× magnification changer allows a native 20Xobjective lens 600 to scan the sample 590 at 40× magnification.

The line scan camera 615 comprises at least one linear array of pictureelements (“pixels”). The line scan camera may be monochrome or color.Color line scan cameras typically have at least three linear arrays,while monochrome line scan cameras may have a single linear array orplural linear arrays. Any type of singular or plural linear array,whether packaged as part of a camera or custom-integrated into animaging electronic module, can also be used. For example, 3 linear array(“red-green-blue” or “RGB”) color line scan camera or a 96 linear arraymonochrome TDI may also be used. TDI line scan cameras typically providea substantially better signal-to-noise ratio (“SNR”) in the outputsignal by summing intensity data from previously imaged regions of aspecimen, yielding an increase in the SNR that is in proportion to thesquare-root of the number of integration stages. TDI line scan camerascomprise multiple linear arrays, for example, TDI line scan cameras areavailable with 24, 32, 48, 64, 96, or even more linear arrays. Thescanner system 550 also supports linear arrays that are manufactured ina variety of formats including some with 512 pixels, some with 1024pixels, and others having as many as 4096 pixels. Similarly, lineararrays with a variety of pixel sizes can also be used in the scannersystem 550. The salient requirement for the selection of any type ofline scan camera 615 is that the motion of the stage 580 can besynchronized with the line rate of the line scan camera 615 so that thestage 580 can be in motion with respect to the line scan camera 615during the digital image capture of the sample 590.

The image data generated by the line scan camera 615 is stored a portionof the memory 565 and processed by the processor 555 to generate acontiguous digital image of at least a portion of the sample 590. Thecontiguous digital image can be further processed by the processor 555and the revised contiguous digital image can also be stored in thememory 565.

In an embodiment with two or more line scan cameras 615, at least one ofthe line scan cameras 615 can be configured to function as a focusingsensor that operates in combination with at least one of the line scancameras that is configured to function as an imaging sensor. Thefocusing sensor can be logically positioned on the same optical path asthe imaging sensor or the focusing sensor may be logically positionedbefore or after the imaging sensor with respect to the scanningdirection of the scanner system 550. In such an embodiment with at leastone line scan camera 615 functioning as a focusing sensor, the imagedata generated by the focusing sensor is stored a portion of the memory565 and processed by the one or more processors 555 to generate focusinformation to allow the scanner system 550 to adjust the relativedistance between the sample 590 and the objective lens 600 to maintainfocus on the sample during scanning.

In operation, the various components of the scanner system 550 and theprogrammed modules stored in memory 565 enable automatic scanning anddigitizing of the sample 590, which is disposed on a glass slide 585.The glass slide 585 is securely placed on the movable stage 580 of thescanner system 550 for scanning the sample 590. Under control of theprocessor 555, the movable stage 580 accelerates the sample 590 to asubstantially constant velocity for sensing by the line scan camera 615,where the speed of the stage is synchronized with the line rate of theline scan camera 615. After scanning a stripe of image data, the movablestage 580 decelerates and brings the sample 590 to a substantiallycomplete stop. The movable stage 580 then moves orthogonal to thescanning direction to position the sample 590 for scanning of asubsequent stripe of image data, e.g., an adjacent stripe. Additionalstripes are subsequently scanned until an entire portion of the sample590 or the entire sample 590 is scanned.

For example, during digital scanning of the sample 590, a contiguousdigital image of the sample 590 is acquired as a plurality of contiguousfields of view that are combined together to form an image strip. Aplurality of adjacent image strips are similarly combined together toform a contiguous digital image of a portion or the entire sample 590.The scanning of the sample 590 may include acquiring vertical imagestrips or horizontal image strips. The scanning of the sample 590 may beeither top-to-bottom, bottom-to-top, or both (bi-directional) and maystart at any point on the sample. Alternatively, the scanning of thesample 590 may be either left-to-right, right-to-left, or both(bi-directional) and may start at any point on the sample. Additionally,it is not necessary that image strips be acquired in an adjacent orcontiguous manner. Furthermore, the resulting image of the sample 590may be an image of the entire sample 590 or only a portion of the sample590.

In one embodiment, computer-executable instructions (e.g., programmedmodules and software) are stored in the memory 565 and, when executed,enable the scanning system 550 to perform the various functionsdescribed herein. In this description, the term “computer-readablestorage medium” is used to refer to any media used to store and providecomputer executable instructions to the scanning system 550 forexecution by the processor 555. Examples of these media include memory565 and any removable or external storage medium (not shown)communicatively coupled with the scanning system 550 either directly orindirectly, for example via a network (not shown).

FIG. 12B illustrates a line scan camera having a single linear array640, which may be implemented as a charge coupled device (“CCD”) array.The single linear array 640 comprises a plurality of individual pixels645. In the illustrated embodiment, the single linear array 640 has 4096pixels. In alternative embodiments, linear array 640 may have more orfewer pixels. For example, common formats of linear arrays include 512,1024, and 4096 pixels. The pixels 645 are arranged in a linear fashionto define a field of view 625 for the linear array 640. The size of thefield of view varies in accordance with the magnification of the scannersystem 550.

FIG. 12C illustrates a line scan camera having three linear arrays, eachof which may be implemented as a CCD array. The three linear arrayscombine to form a color array 650. In one embodiment, each individuallinear array in the color array 650 detects a different color intensity,for example red, green, or blue. The color image data from eachindividual linear array in the color array 650 is combined to form asingle field of view 625 of color image data.

FIG. 12D illustrates a line scan camera having a plurality of lineararrays, each of which may be implemented as a CCD array. The pluralityof linear arrays combine to form a TDI array 655. Advantageously, a TDIline scan camera may provide a substantially better SNR in its outputsignal by summing intensity data from previously imaged regions of aspecimen, yielding an increase in the SNR that is in proportion to thesquare-root of the number of linear arrays (also referred to asintegration stages). A TDI line scan camera may comprise a largervariety of numbers of linear arrays, for example common formats of TDIline scan cameras include 24, 32, 48, 64, 96, 120 and even more lineararrays.

EXAMPLE EMBODIMENTS

The disclosure of the present application may be embodied in a systemcomprising a non-transitory computer readable medium configured to storedata and executable programmed modules; at least one processorcommunicatively coupled with the non-transitory computer readable mediumand configured to execute instructions stored thereon; a register modulestored in the non-transitory computer readable medium and configured tobe executed by the processor, the register module configured to obtain afirst digital image of a specimen in XYZ color, the first digital imagehaving a plurality of image pixels having a first image pixel size,obtain a second digital image of the specimen in RGB color, the seconddigital image having a plurality of image pixels having a second imagepixel size, convert the first digital image and the second digital imageto a common image pixel size, and align the converted image pixels ofthe first digital image with corresponding converted image pixels of thesecond digital image. Such a system may be implemented as a processorenabled device such as the digital imaging device or the imageprocessing device previously described with respect to FIGS. 4A and12A-12D.

The disclosure of the present application may also be embodied in asystem comprising a non-transitory computer readable medium configuredto store data and executable programmed modules; at least one processorcommunicatively coupled with the non-transitory computer readable mediumand configured to execute instructions stored thereon; a register modulestored in the non-transitory computer readable medium and configured tobe executed by the processor, the register module configured to: obtaina first digital image of a specimen in XYZ color, the first digitalimage having a plurality of image pixels having a first image pixelsize, obtain a second digital image of the specimen in RGB color, thesecond digital image having a plurality of image pixels having a secondimage pixel size, align image data of the first digital image withcorresponding image data of the second digital image, and convert thefirst digital image and the second digital image to a common image pixelsize, wherein the converted image pixels of the first digital image arealigned with corresponding converted image pixels of the second digitalimage; a look up table module stored in the non-transitory computerreadable medium and configured to be executed by the processor, the lookup table module configured to: generate a look up table to associate theXYZ color values of a plurality of converted image pixels of the firstdigital image with the RGB color values of a plurality of correspondingconverted image pixels of the second digital image. Such a system may beimplemented as a processor enabled device such as the digital imagingdevice or the image processing device previously described with respectto FIGS. 4A and 12A-12D.

The disclosure of the present application may be embodied in a systemcomprising a non-transitory computer readable medium configured to storedata and executable programmed modules; at least one processorcommunicatively coupled with the non-transitory computer readable mediumand configured to execute instructions stored thereon; a register modulestored in the non-transitory computer readable medium and configured tobe executed by the processor, the register module configured to: obtaina first digital image of a specimen in XYZ color, the first digitalimage having a plurality of image pixels having a first image pixelsize, obtain a second digital image of the specimen in RGB color, thesecond digital image having a plurality of image pixels having a secondimage pixel size, align image data of the first digital image withcorresponding image data of the second digital image, and convert thefirst digital image and the second digital image to a common image pixelsize, wherein the converted image pixels of the first digital image arealigned with corresponding converted image pixels of the second digitalimage; a color module stored in the non-transitory computer readablemedium and configured to be executed by the processor, the color moduleconfigured to: identify a first set of image pixels in the first digitalimage, wherein each image pixel in the first set of image pixels in thefirst digital image has substantially a same XYZ color value, determinean average XYZ color value for the first set of image pixels in thefirst digital image, identify a second set of image pixels in the seconddigital image, wherein each image pixel in the second set of imagepixels in the second digital image corresponds to an image pixel in thefirst set of image pixels in the first digital image, and determine anaverage RGB color value for the second set of image pixels in the seconddigital image; a look up table module stored in the non-transitorycomputer readable medium and configured to be executed by the processor,the look up table module configured to: generate a look up table toassociate the average XYZ color value of the first set of image pixelsin the first digital image with the average RGB color value of thecorresponding second set of image pixels in the second digital image.Such a system may be implemented as a processor enabled device such asthe digital imaging device or the image processing device previouslydescribed with respect to FIGS. 4A and 12A-12D.

Any of the three system embodiments described above may further embodywherein each of the image pixels in the first set of image pixels in thefirst digital image is contiguous with at least one other image pixel inthe first set of image pixels in the first digital image.

Alternatively, any of the three system embodiments described above mayfurther embody wherein at least some of the image pixels in the firstset of image pixels in the first digital image are not contiguous, andfurthermore, wherein the color module is further configured to identifya plurality of first sets of image pixels in the first digital image,wherein each image pixel in a first set of image pixels in the firstdigital image has substantially a same XYZ color value.

The disclosure of the present application may also be embodied in atechnical system comprising a non-transitory computer readable mediumconfigured to store executable programmed modules and at least oneprocessor communicatively coupled with the non-transitory computerreadable medium configured to execute instructions to perform stepscomprising: scanning a specimen using a hyperspectral imaging system togenerate a first digital image of the specimen in XYZ color; scanningthe same specimen using a digital pathology imaging system to generate asecond digital image of the specimen in RGB color; registering the firstdigital image to the second digital image to align the image data in thefirst digital image and the second digital image; generating a lookuptable that associates the XYZ color of the first digital image and theRGB color of the second digital image. Such a system may be implementedas a processor enabled device such as the digital imaging device or theimage processing device previously described with respect to FIGS. 4Aand 12A-12D.

This system embodiment may further include providing XYZ color data to adisplay module for presentation of the second digital image on adisplay.

This system embodiment may further include storing the XYZ color data aspart of the second digital image.

This system embodiment may further include using pattern matching toregister the first digital image to the second digital image.

This system embodiment may further include overlaying a grid on thespecimen prior to creating the first and second digital images and usingthe grid in the first and second digital images to register the firstdigital image to the second digital image.

This system embodiment may further include combining pixels in one ormore of the first digital image and the second digital image to causethe pixel size in the first digital image to be substantially the sameas the pixel size in the second digital image.

This system embodiment may further include generating a single lookuptable for a single stain.

This system embodiment may further include generating a single lookuptable for a plurality of stains.

The disclosure of the present application may also be embodied in atechnical system comprising a non-transitory computer readable mediumconfigured to store executable programmed modules and at least oneprocessor communicatively coupled with the non-transitory computerreadable medium configured to execute instructions to perform stepscomprising: obtaining a first digital image of a specimen scanned by afirst imaging system to generate the first digital image of the specimenin XYZ color, the first digital image having a plurality of image pixelshaving a first image pixel size; obtaining a second digital image of thespecimen scanned by a second imaging system to generate the seconddigital image of the specimen in RGB color, the second digital imagehaving a plurality of image pixels having a second image pixel size;registering the first digital image to the second digital image to alignthe image data in the first digital image and the second digital image;presenting the second digital image on a display; using a colormeasurement device to measure the XYZ values of the color presented onthe display in a first region; and comparing the measured XYZ values ofthe first region to the XYZ values of the first digital image for thefirst region to validate the digital pathology system. Such a system maybe implemented as a processor enabled device such as the digital imagingdevice or the image processing device previously described with respectto FIGS. 4A and 12A-12D.

The disclosure of the present application may also be embodied in amethod comprising: obtaining a first digital image of a specimen scannedby a first imaging system to generate the first digital image of thespecimen in XYZ color, the first digital image having a plurality ofimage pixels having a first image pixel size, obtaining a second digitalimage of the specimen scanned by a second imaging system to generate thesecond digital image of the specimen in RGB color, the second digitalimage having a plurality of image pixels having a second image pixelsize, generating a lookup table to associate the XYZ color of the firstdigital image and the RGB color of the second digital image. Such amethod may be implemented by a system such as the digital imaging deviceor the image processing device previously described with respect toFIGS. 4A and 12A-12D.

This method embodiment may further include aligning image data in thefirst digital image with image data in the second digital image; andgenerating a lookup table to associate the XYZ color of the firstdigital image with the corresponding RGB color of the second digitalimage in accordance with said alignment.

This method embodiment may further include, wherein the first digitalimage comprises a plurality of image pixels having a first image pixelsize and the second digital image comprises a plurality of image pixelshaving a second image pixel size, converting the image pixels of thefirst digital image and the image pixels of the second digital image toa common image pixel size, and aligning the image pixels of theconverted first digital image with corresponding image pixels of theconverted second digital image.

This method embodiment may further include identifying a first set ofimage pixels in the first digital image, wherein each image pixel in thefirst set of image pixels in the converted first digital image hassubstantially a same XYZ color value; determining an average XYZ colorvalue for the first set of image pixels in the converted first digitalimage; identifying a second set of image pixels in the converted seconddigital image, wherein each image pixel in the second set of imagepixels in the converted second digital image corresponds to an imagepixel in the first set of image pixels in the converted first digitalimage; determining an average RGB color value for the second set ofimage pixels in the converted second digital image, and generating alookup table to associate the average XYZ color value for the first setof image pixels in the converted first digital image with thecorresponding average RGB color value of the second set of image pixelsin the converted second digital image.

The disclosure of the present application may also be embodied in amethod comprising: obtaining a first digital image of a specimen scannedby a first imaging system to generate the first digital image of thespecimen in XYZ color, the first digital image having a plurality ofimage pixels having a first image pixel size, obtaining a second digitalimage of the specimen scanned by a second imaging system to generate thesecond digital image of the specimen in RGB color, the second digitalimage having a plurality of image pixels having a second image pixelsize, aligning image data in the first digital image with image data inthe second digital image; and generating a lookup table to associate theXYZ color of the first digital image with the corresponding RGB color ofthe second digital image in accordance with said alignment. Such amethod may be implemented by a system such as the digital imaging deviceor the image processing device previously described with respect toFIGS. 4A and 12A-12D.

This method embodiment may further include, wherein the first digitalimage comprises a plurality of image pixels having a first image pixelsize and the second digital image comprises a plurality of image pixelshaving a second image pixel size, converting the first digital image andthe second digital image to a common image pixel size, and aligning theconverted image pixels of the first digital image with correspondingconverted image pixels of the second digital image.

This method embodiment may further include identifying a plurality offirst sets of image pixels in the converted first digital image, whereineach image pixel in each set of image pixels in the plurality of firstsets of image pixels in the converted first digital image hassubstantially a same XYZ color value; determining an average XYZ colorvalue for each set of image pixels in the plurality of first sets ofimage pixels in the converted first digital image; identifying acorresponding plurality of second sets of image pixels in the convertedsecond digital image, wherein each image pixel in each set of imagepixels in the plurality of second sets of image pixels in the convertedsecond digital image corresponds to an image pixel in the convertedfirst digital image; determining an average RGB color value for each setof image pixels in the plurality of second sets of image pixels in theconverted second digital image, and generating a lookup table toassociate the average XYZ color value for each of the first sets ofimage pixels in the converted first digital image with the correspondingaverage RGB color value of the corresponding second set of image pixelsin the converted second digital image.

The disclosure of the present application may also be embodied in amethod comprising: obtaining a first digital image of a specimen scannedby a first imaging system to generate the first digital image of thespecimen in XYZ color, the first digital image having a plurality ofimage pixels having a first image pixel size, obtaining a second digitalimage of the specimen scanned by a second imaging system to generate thesecond digital image of the specimen in RGB color, the second digitalimage having a plurality of image pixels having a second image pixelsize, registering the first digital image to the second digital image toalign the image data in the first digital image and the second digitalimage; and generating a lookup table that associates the XYZ color ofthe first digital image and the RGB color of the second digital image.Such a method may be implemented by a system such as the digital imagingdevice or the image processing device previously described with respectto FIGS. 4A and 12A-12D.

This method embodiment may further include providing XYZ color data to adisplay module for presentation of the second digital image on adisplay.

This method embodiment may further include storing the XYZ color data aspart of the second digital image file.

This method embodiment may further include using pattern matching toregister the first digital image to the second digital image.

This method embodiment may further include overlaying a grid on thespecimen prior to creating the first and second digital images and usingthe grid in the first and second digital images to register the firstdigital image to the second digital image.

This method embodiment may further include combining pixels in one ormore of the first digital image and the second digital image to causethe first image pixel size in the first digital image to besubstantially the same as the second image pixel size in the seconddigital image.

This method embodiment may further include generating a single lookuptable for a single stain.

This method embodiment may further include generating a single lookuptable for a plurality of stains.

The disclosure of the present application may also be embodied in amethod comprising: obtaining a first digital image of a specimen scannedby a first imaging system to generate the first digital image of thespecimen in XYZ color, the first digital image having a plurality ofimage pixels having a first image pixel size, obtaining a second digitalimage of the specimen scanned by a second imaging system to generate thesecond digital image of the specimen in RGB color, the second digitalimage having a plurality of image pixels having a second image pixelsize, registering the first digital image to the second digital image toalign the image data in the first digital image and the second digitalimage; presenting the second digital image on a display; using a colormeasurement device to measure the XYZ values of the color presented onthe display in a first region; and comparing the measured XYZ values ofthe first region to the XYZ values of the first digital image for thefirst region to validate the digital pathology system. Such a method maybe implemented by a system such as the digital imaging device or theimage processing device previously described with respect to FIGS. 4Aand 12A-12D.

The disclosure of the present application may also be embodied in amethod comprising: obtaining a first digital image of a specimen scannedby a first imaging system to generate the first digital image of thespecimen in XYZ color, the first digital image having a plurality ofimage pixels having a first image pixel size; obtaining a second digitalimage of the specimen scanned by a second imaging system to generate thesecond digital image of the specimen in RGB color, the second digitalimage having a plurality of image pixels having a second image pixelsize, registering the first digital image to the second digital image toalign the image data in the first digital image and the second digitalimage; converting the first digital image and the second digital imageto a common image pixel size; identifying a plurality of first sets ofimage pixels in the converted first digital image, wherein each imagepixel in each set of image pixels in the plurality of first sets ofimage pixels in the converted first digital image has substantially asame XYZ color value; determining an average XYZ color value for eachset of image pixels in the plurality of first sets of image pixels inthe converted first digital image; identifying a corresponding pluralityof second sets of image pixels in the converted second digital image,wherein each image pixel in each set of image pixels in the plurality ofsecond sets of image pixels in the converted second digital imagecorresponds to an image pixel in the converted first digital image;determining an average RGB color value for each set of image pixels inthe plurality of second sets of image pixels in the converted seconddigital image; generating a lookup table to associate the average XYZcolor value for each of the first sets of image pixels in the convertedfirst digital image with the corresponding average RGB color value ofthe corresponding second set of image pixels in the converted seconddigital image; presenting a first average RGB color value from thelookup table on a first region of a display; using a color measurementdevice to measure the XYZ value from the first region of the display;and comparing the measured XYZ value from the first region of thedisplay to the average XYZ value corresponding to the first average RGBcolor value in the lookup table. Such a method may be implemented by asystem such as the digital imaging device or the image processing devicepreviously described with respect to FIGS. 4A and 12A-12D.

The above description of the disclosed embodiments is provided to enableany person skilled in the art to make or use the invention. Variousmodifications to these embodiments will be readily apparent to thoseskilled in the art, and the generic principles described herein can beapplied to other embodiments without departing from the spirit or scopeof the invention. Thus, it is to be understood that the description anddrawings presented herein represent a presently preferred embodiment ofthe invention and are therefore representative of the subject matterwhich is broadly contemplated by the present invention. It is furtherunderstood that the scope of the present invention fully encompassesother embodiments that may become obvious to those skilled in the artand that the scope of the present invention is accordingly not limited.

What is claimed is:
 1. A method comprising: obtaining a first digitalimage of a specimen in XYZ color captured by a first imaging system,wherein the specimen is a tissue specimen, wherein the first digitalimage comprises a plurality of XYZ pixels, and wherein each of theplurality of XYZ pixels has an XYZ color value; obtaining a seconddigital image of the specimen in RGB color captured by a second imagingsystem, wherein the second imaging system is separate from the firstimaging system, wherein the second digital image comprises a pluralityof RGB pixels, and wherein each of the plurality of RGB pixels has anRGB color value; and generating a lookup table to associate each of aplurality of XYZ color values derived from the first digital image withone of a plurality of RGB color values derived from the second digitalimage.
 2. The method of claim 1, wherein generating the lookup tablecomprises: registering the first digital image and the second digitalimage to a common grid; and mapping the plurality of XYZ pixels to theplurality of RGB pixels according to the common grid.
 3. The method ofclaim 1, wherein generating the lookup table comprises: aligning thefirst digital image with the second digital image based on patternmatching; and mapping the plurality of XYZ pixels to the plurality ofRGB pixels according to the alignment.
 4. The method of claim 1, furthercomprising up-sampling or down-sampling one or both of the first digitalimage and the second digital image to a common pixel size.
 5. The methodof claim 1, wherein the plurality of XYZ color values derived from thefirst digital image are average XYZ color values for subsets of theplurality of XYZ pixels, and wherein the plurality of RGB color valuesderived from the second digital image are average RGB color values forsubsets of the plurality of RGB pixels.
 6. The method of claim 5,further comprising: determining the subsets of XYZ pixels by groupingthose of the plurality of XYZ pixels that have XYZ color values within aXYZ threshold; and determining the subsets of RGB pixels by groupingthose of the plurality of RGB pixels that have RGB color values within aRGB threshold.
 7. The method of claim 6, wherein an amount of thesubsets of XYZ pixels and an amount of the subsets of RGB pixels areboth limited to a predefined number, such that an amount of theplurality of XYZ color values and an amount of the plurality of RGBcolor values in the lookup table are also both limited to the predefinednumber.
 8. The method of claim 7, wherein the predefined number is ten.9. The method of claim 6, wherein an amount of the subsets of XYZ pixelsand an amount of the subsets of RGB pixels are limited to a smallestnumber that allocates each of the plurality of XYZ pixels to one of theplurality of XYZ color values and allocates each of the plurality of RGBpixels to one of the plurality of RGB color values, while minimizing anerror between the respective color values of the pixels and therespective color values to which the pixels are allocated.
 10. Themethod of claim 9, wherein the error is minimized using a root meansquare analysis.
 11. The method of claim 1, further comprising: for eachof a plurality of contiguous sets of two or more of the plurality of XYZpixels that have similar XYZ color values, grouping the contiguous setof two or more XYZ pixels into an XYZ superpixel, and assigning anaverage XYZ color value of the contiguous set of two more XYZ pixels tothe XYZ superpixel; and, for each of a plurality of contiguous sets oftwo or more of the plurality of RGB pixels that have similar RGB colorvalues, grouping the contiguous set of two or more RGB pixels into anRGB superpixel, and assigning an average RGB color value of thecontiguous set of two more RGB pixels to the RGB superpixel; wherein theplurality of XYZ color values is derived from the average XYZ colorvalues assigned to the XYZ superpixels, and wherein the plurality of RGBcolor values is derived from the average RGB color values assigned tothe RGB superpixels.
 12. The method of claim 1, wherein capturing thefirst digital image of a specimen in XYZ color via the first imagingsystem comprises: capturing a hyperspectral image stack comprisingimages of the specimen at different wavelengths of light; and generatingthe first digital image from the hyperspectral image stack.
 13. Themethod of claim 1, further comprising embedding the lookup table into adata structure that comprises the second digital image.
 14. The methodof claim 13, wherein the lookup table is comprised in an InternationalColor Consortium (ICC) profile in the data structure.
 15. The method ofclaim 1, further comprising converting the lookup table into a model,and embedding the model into a data structure that comprises the seconddigital image.
 16. A system comprising: at least one hardware processor;and one or more software modules that are configured to, when executedby the at least one hardware processor, obtain a first digital image ofa specimen in XYZ color captured by a first imaging system, wherein thefirst digital image comprises a plurality of XYZ pixels, and whereineach of the plurality of XYZ pixels has an XYZ color value, obtain asecond digital image of the specimen in RGB color captured by a secondimaging system, wherein the second imaging system is separate from thefirst imaging system, wherein the second digital image comprises aplurality of RGB pixels, and wherein each of the plurality of RGB pixelshas an RGB color value, and generate a lookup table to associate each ofa plurality of XYZ color values derived from the first digital imagewith one of a plurality of RGB color values derived from the seconddigital image.
 17. A non-transitory computer-readable medium havinginstructions stored thereon, wherein the instructions, when executed bya processor, cause the processor to: obtain a first digital image of aspecimen in XYZ color captured by a first imaging system, wherein thefirst digital image comprises a plurality of XYZ pixels, and whereineach of the plurality of XYZ pixels has an XYZ color value; obtain asecond digital image of the specimen in RGB color captured by a secondimaging system, wherein the second imaging system is separate from thefirst imaging system, wherein the second digital image comprises aplurality of RGB pixels, and wherein each of the plurality of RGB pixelshas an RGB color value; and generate a lookup table to associate each ofa plurality of XYZ color values derived from the first digital imagewith one of a plurality of RGB color values derived from the seconddigital image.